Reproducible Assessment of General Education

Systematic Use of Course-Level Student Data

Dr. Clifton Franklund

October 24, 2017

Presentation Handouts

  • PDF versions are available online

About Ferris State

  • West Central Michigan
  • Medium Masters University
  • 14,600 students
  • Recently Revised General Education
  • Opportunity to embed assessment

General Education Assessment

  • Goal: Track and improve program effectiveness

    • Our program primarily consists of courses
    • Embedded course assignments are the most meaningful and valid measures of student learning.
    • General Education applies to all students
  • Therefore, we are measuring student performance in General Education using embedded assessments for all students in all courses.

Project Scope

  • In a typical semester, General Education at Ferris:

    • Offers over 400 different courses
    • Instructs over 5,000 individual students
    • Involves over 500 different faculty
    • Generates over 50,000 student credit hours
    • Is responsible for $20,000,000 in revenue

Project Resources

  • Program coordinator (me, 50% release)
  • General Education Committee (advisory, no release)
  • Line item budget ($0)
  • Dedicated staff (none)
  • Other staff (Registrar and IR)

The Assessment Cycle

The Futile Cycle

A Better Model

“Assessment is not a spreadsheet – it is a conversation”
Irmeli Halinen

Reproducible assessment

Our goal is to create reproducible processes to increase transparency and encourage faculty engagement.
  • ALL assessment resources are public-facing

    • Data reports (PDF and HTML)
    • Data analysis code (R)
    • Deidentified data files (csv)
  • All released with MIT or CC-BY licenses
  • All results are computationally reproducible and extensible

Striking a balance

Faculty ownership vs. Chaotic data catastrophe

Sustainable assessment

  • The three keys to our program assessment strategy:

  1. Simplification — reduce complexity
  2. Standardization — reduce variance
  3. Automation — reduce workload

Simplify: Core Competencies

  • Eight competencies:
  • Collaboration, Communication, Culture, Diversity, Natural Sciences, Problem Solving, Quantitative Literacy, Self and Society
  • Each overseen by a faculty subcommittee (6-10 members)

  1. Operational definitions
  2. Goals (Hallmarks of a Bulldog)
  3. Relevance statements

Simplify: Ferris Learning Outcomes

  • Each competency has four learning outcomes
  • Based upon what we want our graduating seniors to look like
  • Derived from LEAP
  • One outcome measured each semester
  • Rolling two-year program cycle

Standardize: Rubrics

  1. Scoring system (zero to four point scale)
  2. Analysis interpretation (four threshold values)
  3. Criterion leveling (expectations by course level)

Standardize: Measures

  • Created in order to get more uniform data collection and summarization
  • 14 different type of measures are defined
  • Instructors can choose to use any one they wish

  1. Exams
  2. Products
  3. Performances

Standardize: Registering Courses

  • I need to know who is doing what each semester
  • Who is collecting data?
  • What competency is being measured?
  • What outcome is being measured?
  • What measure is being used?
  • All tracked using a Google form

Standarize: Data Collection

  • Excel workbooks to collect and summarize student results
  • SCORE, PREPOST, NORM, RUBRIC
  • Common metadata supplied by instructors
  • Enter student names, IDs, and appropriate evaluations
  • A standaridized summary is automatically generated

Standardize: TracDat Assignments

  • Seeking to minimize the need for faculty to log into TracDat
  • Received as an email
  • Copy and paste summary
  • Select a few responses from drop-down menus
  • Add a short reflection
  • Attach the workbook and submit

Automate: Analysis of Student Data

  • All workbooks are stored in the TracDat document repository
  • Each semester’s data is in a separate folder
  • Download all the workbooks
  • An R script is used to process the workbooks
  • All data is aggregated into one csv file
  • Identifiable information is removed

Automate: Reporting

  • Using R and bookdown (Rmarkdown)
  • Reuseable, version-controlled code
  • PDF reports for printing
  • HTML reports for distribution and collaboration

Assessment in Action

  • Five means of creating a dialog:

  1. Print reports for dissemination
  2. Online forums for extended discussions
  3. OSF projects as the ultimate home base
  4. GitHub repositories for reproducibility
  5. Executive dashboard for overviews

Summary

  • We are creating a collaborative assessment environment
  • This is more about process than personality
  • Focus is on making a resource rather than just reports

Questions?

  • All of these resources are linked to from your handouts
  • Licensed using either CC-BY 4.0 or MIT
  • Feel free to contact me if you have questions or comments